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ABSTRACT 

In  this  paper,  we  propose  three  new  proportionate-type  NLMS  al¬ 
gorithms:  the  water  filling  algorithm,  the  feasible  water  filling  algo¬ 
rithm,  and  the  adaptive  p- law  proportionate  NLMS  (MPNLMS)  al¬ 
gorithm.  The  water  filling  algorithm  attempts  to  choose  the  optimal 
gains  at  each  time  step.  The  optimal  gains  are  found  by  minimizing 
the  mean  square  error  (MSE)  at  each  time  with  respect  to  the  gains, 
given  the  previous  mean  square  weight  deviations.  While  this  algo¬ 
rithm  offers  superior  convergence  times,  it  is  not  feasible.  The  sec¬ 
ond  algorithm  is  a  feasible  version  of  the  water  filling  algorithm.  The 
adaptive  MPNLMS  (AMPNLMS)  algorithm  is  a  modification  of  the 
MPNLMS  algorithm.  In  the  MPNLMS  algorithm,  the  parameter  p 
of  the  ^t-law  compression  is  constant.  In  the  AMPNLMS  algorithm 
the  parameter  p  is  allowed  to  vary  with  time.  This  modification  al¬ 
lows  the  algorithm  more  flexibility  when  attempting  to  minimize  the 
MSE.  Compared  with  several  feasible  algorithms,  the  AMPNLMS 
algorithm  has  the  fastest  MSE  decay  for  almost  all  times. 

Index  Terms  —  Adaptive  filters,  least  mean  square  methods, 
convergence  of  numerical  methods,  sparse  impulse  response. 


1.  PTNLMS  ALGORITHM  FRAMEWORK 

Beginning  with  the  proportionate  normalized  least  mean 
square  (PNLMS)  algorithm  [1],  proportionate-type  NLMS  (PtNLMS) 
algorithms  have  been  widely  used  in  adaptive  filtering  applications 
when  dealing  with  sparse  impulse  responses.  A  sparse  impulse  re¬ 
sponse  is  one  in  which  very  few  coefficients  differ  from  zero. 

In  this  section  we  lay  out  the  framework  of  a  canonical  PtNLMS 
algorithm.  Our  general  goal  is  to  estimate  an  unknown  system  im¬ 
pulse  response  of  length  L  given  by  w  =  [wi,W2,  . . . ,  wl]-  The 
input  signal  to  the  unknown  system  at  time  k  is  x(fc)  =  [x[k),x[k— 
1), . . . ,  x(k—L+l)]T .  The  output  ofthe  system  is  y(k)  =  wTx(fc). 
The  measured  output  of  the  system,  d(k),  contains  noise  v(k)  and  is 
equal  to  the  sum  of  y(k)  and  v(k).  The  algorithm  is  designed  to  es¬ 
timate  the  system  impulse  response.  We  define  the  estimated  system 
impulse  response  as  w (k).  The  estimated  system  output  is  given  by 
y(k )  =  wT(fc)x(fc).  We  measure  the  error,  defined  as  e(fc),  of  our 
estimate  by  taking  the  difference  between  the  measured  output  and 
the  estimated  output. 

The  update  equation  for  the  canonical  PtNLMS  algorithm  is 
given  in  Table  1.  The  S  term  is  introduced  into  the  denominator  to 
avoid  division  by  zero  in  situations  where  the  inputs  signal  has  no 
energy.  In  Table  II  we  specify  the  generation  of  the  time  varying 
gain  matrix.  The  user  starts  by  choosing  an  appropriate  function  of 


Table  I 

PtNLMS  Algorithm 

y{k) 

=  wJ  x(fc) 

y{k) 

=  wTx(fc) 

d{k) 

=  y(k)  +  v(k) 

e{k) 

=  d(k)  -  y(k) 

■w(k  +  1) 

-  -i./'l-'l  |  /3G(k+l)x(k)e(fc) 

WW  "+■  xr(k)Gffc+l)x(fe)+i5 

Table  II 


PtNLMS  Gain  Matrix  Generation 


Specify 

7min  {k  +  1) 

Jl{k  +  1) 

gi{k  + 1) 

G(fc  +  1) 


F[\wi{k)\] 

p  max{<5p,  F  [|u>1(fc)|]  [|w;(fc)|]} 

max{7min(fc  +  l),F[|i&i(fc)|]} 

7l(fc  +  !) 

i  i  7i(fc+i) 

diag{gi(fc  +  1 gL[k  +  1)} 


the  estimated  impulse  response  F[|t&((fc)|],  l  =  1,  2,  ■  ■  ■  ,  L.  For  in¬ 
stance  if  we  specify  F[|wii(fc)|]  =  1  the  algorithm  becomes  the  well 
known  NLMS  algorithm.  Next  a  minimum  gain  7min(A;  +  1)  is  cal¬ 
culated.  The  minimum  gain  ensures  that  all  coefficients  are  updated 
at  each  iteration.  The  constant  5P  is  important  in  the  beginning  of  the 
algorithm  when  all  of  the  coefficients  are  zero  and  together  with  p 
prevents  the  very  small  coefficients  from  stalling. 

2.  RECURSIVE  WEIGHT  DEVIATION  EQUATIONS 

The  majority  of  the  analysis  that  follows  is  based  on  the  recursion 
equations  for  the  weight  deviation  and  the  square  weight  deviation. 
In  this  section  we  present  these  recursion  equations  in  vector  and 
component-wise  fonn.  The  weight  deviation  at  time  k  is  defined  as 
z (fc)  =  w  —  w(fc).  The  update  equation  for  the  weight  deviation 
vector  is 


z (k  +  1) 


.  .  /3G(k  +  l)x(fc)xT(fc)z(fc) 

x(fc)G(fc  +  l)x(fc)  +  S 
/3G  (k  +  1  )x(fc)u(fc) 
x.T(k)G(k  +  l)x(fc)  +  S 


Employing  the  convention  that  Xi(k)  is  the  ith  component  of  vector 
x(fc),  we  can  express  (1 )  and  the  square  weight  deviation  [z?  (k  + 1)] 
in  component-wise  form  as  follows: 
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Zi(k  +  1) 


=  Zi(k)  - 


Pgi{k  +  1  )xj(k)  xj(k)zj(k) 

ELi  x2Ak)gj(k+\)  +  S 


Pgpk  +  l)xj(k)v(k) 
ELi  x?(k)gj(k  +  l)  +  5 


(2) 


Zi{k  +  1)  =  zf(k) 

02gj{k  +  l)xj(k)  Ej=i  E^Li  Xj(k)xi(k)zj(k)zi{k) 
[Et=i  x2(k)9j{k  +  1)  +5] 
f32gUk  +  l)xUk)v2(k) 

(Et=i^2(%,(fc  +  i)  +  <5)2 

2j3gi(k  +  l)zi(fc)  E^=i  xj(k)zj(k)zi(k) 

T,f=i  x2j(k)9j{k  +  l)  +  5 
2  Pgi{k  +  1  )xi(k)v(k)zi(k) 

T,f=i  x2(k)gj(k+l)  +  S 

W2g2ik  +  !)»?(&)  EjL=i  xj(k)zj(k)v(k ) 

+ - F — 7 - ' - T2 - ■  ( 

[Ej=i  x2(k)9i(k  +  1)  +  <5j 

3.  OPTIMAL  PTNLMS  ALGORITHM 

We  begin  by  making  the  following  assumptions. 


zero 

reference 

level 


Fig.  1.  Water  filling  solution  for  A. 


and  the  constraint  Ey=i  9i(k  +  1)  =  L.  Note  that  J(k)  is  the 
approximate  MSE  at  instant  k  [3],  We  can  recast  this  problem  as  an 
optimization  problem  in  the  form 

min  J  (k  +  1)  =  cTg  +  ^gTQg  +  <r2 

g  2 

gi  >  0  V  i 

lTg  =  L  (7) 


Assumption  I:  The  input  signal  is  a  white  Gaussian  noise  process 
with  zero-mean  and  variance  a 

Assumption  II:  /3  is  sufficiently  small  such  that  Zi(k)  fluctuates 
much  slower  than  x(fc)  and  therefore  they  can  he  considered  inde¬ 
pendent. 

Assumption  III:  The  denominator  terms  are  assumed  to  be  constant. 
(The  validity  of  this  assumption  was  discussed  in  [2].) 

Assumption  IV:  The  measurement  noise  v(k)  is  white  with  zero- 
mean,  variance  and  it  is  independent  of  the  input. 

Under  these  assumptions  and  assuming  that  gi{k  +  1)  is  a  deter¬ 
ministic  function  of  time,  we  arrive  at  the  following  equations  after 
taking  the  expectation  of  the  weight  deviation  and  the  square  weight 
deviation: 


E{zi{k  1)}  = 

E{zf(k  +  1)}  = 


+ 


+ 


Zi{k  +  1)  =  Zi(k )  -  (3ogi{k  +  l)zi(k)  (4) 
z?(k  +  1)  =  z?(k)  -  2f3ogi(k  +  1  )zf(k) 


Po  9i  (k  +  1) 


3  +  £>“(*) 


P20^g2(k  +  1). 


(5) 


Here,  we  have  defined  /3o  =  Pa^/iLa^+S).  We  make  the  following 
definitions  for  notational  convenience: 


a  =  -2p0zf(k) 


Qi  = 


2/?o 


3 z?(k)  +  ^z?(k)  J  +  ^ 


(6) 


At  this  point  we  try  to  find  the  optimal  gain  by  minimizing,  J{k  + 
1)  =  <r%  +  cr2  EiLi  zi(k  +  1)  for  all  k,  with  respect  to  the  gain 


where  g  =  [g^k  +  l),g2(fc  +  1 gL(k  +  1)]T  and  Q  = 
diag(gi,?2,  ■  ■ 

Next,  for  convenience,  we  make  the  substitution  gi(k  +  1)  =  sf 
and  incorporate  the  constraint  into  the  minimization  problem.  This 
results  in 

L  L  L 

T(s,  A)  =  dSi  +  —  qiSi  +  A(^^  Si  —  L).  (8) 

2  =  1  2=1  2=1 

Taking  the  first  derivative  and  setting  it  equal  to  zero  yields  two  solu¬ 
tions,  Sk  =  0  and  sk  =  ■ \/(—Ck  —  ty/qk-  If  we  examine  the  second 
derivative  JtUU  we  find  that  the  first  solution  results  in  a  minimum  if 
A  >  — Cfe  and  the  second  solution  results  in  a  minimum  if  A  <  —Ck- 
As  it  turns  out,  the  solution  to  this  problem  is  of  the  “water  filling” 
variety.  We  choose  the  constant  A  according  to  the  following  rules. 
First  we  sort  the  entries  of  —  c  in  ascending  order  to  form  a  new  vec¬ 
tor  such  that  — C(i)  <  —  C(2)  <  ...  <  — C(z-).  We  subsequently 
rearrange  the  elements  of  Q  to  match  the  position  of  the  original 
indices  in  the  sorted  —  c  and  to  form  a  new  matrix  whose  diagonal 
elements  are  q(  1),  q(2)  . . .,  q(L)- 
Next  we  solve  for 


_  CW  _  t 

2^k=i  qw 


(9) 


We  choose  A  =  A i  if  — C(4_i)  <  Ai  <  —  c^,  where  — C(0)  =  —00. 
We  depict  this  water-filling  problem  in  Fig.  1.  We  have  a  pitcher 
containing  a  volume  L  of  water.  After  pouring  the  contents  of  the 
pitcher  in  a  vessel  with  the  profile  shown  in  Fig.  1,  we  want  to  know 
the  difference  of  the  height  the  water  reaches  in  the  vessel  and  the 
zero-reference  level  of  the  vessel.  This  difference  gives  us  the  value 
of  A  we  are  seeking.  The  stairway  sloped  bottom  of  the  vessel  is 
defined  by  the  treads  of  heights  —  c^)  and  with  widths,  1  /q(k)- 
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4.  FEASIBLE  SUBOPTIMAL  PTNLMS  ALGORITHM 


LEARNING  CURVE  COMPARISON 


The  algorithm  proposed  in  the  previous  sections  is  not  feasible.  Spe¬ 
cifically  we  need  to  know  E{z2{k)}  =  E{[wi  —  Wi(k)}2},  which 
requires  knowledge  of  the  optimal  impulse  response.  In  order  to 
avoid  this  we  rewrite  the  error  in  another  fonn 

L 

«(*)  =  £  Xj(k)zj(k)  +  v(k).  (10) 

3=1 


Next  we  multiply  both  sides  of  the  equation  by  Xi{k)  and  take  the 
expectation  [assuming  a h{k)  is  a  white  signal].  This  results  in 

E{xi{k)e(k)}  =  a2E{zi(k)}.  (11) 


If  we  define  pi  (k)  =  Xi(k)e(k),  we  can  calculate  p^k)  =  E{pi(k)} 
and  then  solve  for  Zi(k).  We  update  our  estimate  of  Pi(k)  in  the  fol¬ 
lowing  fashion: 


aPi{k-  1)  +  (1  -  a)pi(k) 


Zi{k)  = 


P»(fc) 


(12) 


where  0  <  a  <  1.  Now,  we  approximate  E{z2  (k)}  with  [zi(fc)] 2 . 
This  approximation  is  expected  to  be  good  at  the  beginning  when 
Zi(k)  are  large.  The  same  approximation  is  used  in  [4],  [5].  Better 
estimates  of  E{z2 (fc)}  are  possible,  but  more  complex  to  compute. 


5.  ADAPTIVE  MPNLMS  ALGORITHM 


The  AMPNLMS  algorithm  is  a  modification  of  the  MPNLMS  algo¬ 
rithm  [6],  [7],  As  in  the  previous  section  we  try  to  reach  the  optimal 
perfonnance  as  quickly  as  possible  at  each  time  step.  In  order  to 
do  so  we  introduce  a  time  varying  ec(fc),  such  that  F  [|uh(fc)|]  = 
In  [1  +  fi(k)\wi(k)\]  and  ec(k)  =  l/p(k).  We  recall  that  ec  is  re¬ 
lated  to  when  a  coefficient  is  considered  to  be  converged  [7],  The 
strategy  employed  by  the  AMPNLMS  is  to  start  out  with  a  large 
value  for  ec(fc)  and  slowly  decrease  the  required  e-neighborhood  to 
be  reached  by  the  converged  algorithm.  In  doing  so,  the  AMPNLMS 
algorithm  initially  behaves  like  the  PNLMS  algorithm  and  then  tran¬ 
sitions  to  performing  like  the  NLMS  algorithm  as  time  proceeds. 
This  transition  reflects  our  knowledge  of  the  impulse  response.  Ini¬ 
tially  we  know  the  impulse  response  is  sparse  and  therefore  we  direct 
our  resources  in  a  manner  to  most  efficiently  estimate  the  impulse  re¬ 
sponse.  As  time  goes  on,  our  a  priori  knowledge  [which  is  w(fc)] 
becomes  close  to  the  true  value  of  the  impulse  response  and  the  co¬ 
efficient  estimate  errors  are  unifonnly  distributed  along  all  coeffi¬ 
cients,  in  contrast  to  the  initial  situation. 

We  detennine  the  value  we  choose  for  ec(fc)  by  relating  it  to  the 
current  MSE  e2(k).  We  can  fonn  an  estimate  of  e2(fc)  and  update 
p(k)  by  calculating: 


C(fc  +  l)=££(fc)  +  (l-£)e2(fc) 
iL(k+  1)  =  ^±12 

ec(k  +  1)  = 
p(k  +  1)  = 


£l  (k  T-  1) 
La2 
1 

tc{k  +  1) 


(13) 


where  0  <  £  <  1  and  v  is  a  constant  relating  the  current  MSE 
estimate  (obtained  by  time  averaging)  to  £l  {k  +  1 ) .  The  tenn  C(k  + 


Fig.  2.  Water  filling  a  comparison. 


1)  is  an  estimate  of  e2(k  +  1).  eL(fc  +  1)  is  the  distance  to  the 
steady-state  MSE  (“noise  floor”)  that  is  considered  as  achievement 
of  convergence  when  reached  by  the  MSE. 

In  essence,  the  algorithm  continuously  tightens  the  level  above 
the  noise  floor  that  the  algorithm’s  MSE  is  required  to  achieve.  Pro¬ 
ceeding  in  this  fashion  causes  the  algorithm  to  behave  like  the 
PNLMS  algorithm  initially  when  changing  large  coefficients  results 
in  large  changes  in  the  MSE.  As  time  proceeds  and  the  MSE  bound 
tightens,  coefficients  with  smaller  amplitudes  are  adjusted  until  the 
algorithm  finally  performs  like  the  NLMS  (through  the  ^-compres¬ 
sion  law).  Adaptation  of  fi(k)  is  done  by  using  the  estimate  of  MSE, 
£(fc).  In  this  way,  the  changes  of  p(k)  are  related  in  a  natural  way 
to  the  changes  of  MSE,  i.e.,  when  the  MSE  becomes  smaller,  the 
algorithm  inclines  to  treat  all  coefficients  more  equally. 


6.  RESULTS 

The  parameters  used  to  generate  the  curves  that  follow  are:  L  = 
512,  fi  =  0.1,  p  =  0.01,  <5  =  5.12,  a2x  =  1,  and  cr2  =  10“6.  The 
impulse  response  used  in  simulations  corresponds  to  a  real-world 
network  echo  path  similar  to  the  one  given  in  [6],  [7].  In  Fig.  2 
we  plot  the  MSE  estimate  obtained  by  Monte  Carlo  simulations  for 
the  non-feasible  water  filling  algorithm  and  the  feasible  water  filling 
algorithm  with  values  of  a  =  0.9,  0.99,  and  0.999.  As  expected, 
the  non-feasihle  water  filling  algorithm  perfonns  best.  The  feasible 
water  filling  algorithm  converges  fastest  for  a  =  0.99. 

We  compare  the  estimated  MSE  performance  of  the  AMPNLMS 
using  different  values  of  v  in  Fig.  3.  Specifically,  we  have  chosen 
iv  =  1, 10, 100,  and  1000.  The  value  of  £  =  0.99  is  fixed  in  these 
simulations.  Out  of  four  considered  values  for  v,  the  value  iv=1000 
provides  the  algorithm  with  the  best  performance. 

Next  in  Fig.  4,  we  compare  the  estimated  MSE  performance 
of  the  AMPNLMS  algorithm  for  a  fixed  value  of  v  =  1000,  and 
varying  £  =  0.5,  0.9,  0.99,  and  0.999.  Varying  the  value  of  £  does  not 
have  as  great  an  effect  on  the  MSE  performance  as  the  v  parameter 
does. 

In  Fig.  5  we  compare  the  NLMS,  PNLMS,  MPNLMS,  e-PNLMS 
[8],  AMPNLMS,  feasible  water  filling,  and  water  filling  algorithms. 
The  additional  parameters  used  to  generate  these  curves  are  as  fol¬ 
lows:  a  =  0.99,  £  =  0.99,  and  v  —  1000.  Naturally,  the  non- 
feasible  water-filling  algorithm  performs  better  than  all  of  the  other 
algorithms.  The  AMPNLMS  algorithm  has  the  second  best  perfor- 
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Fig.  3.  AMPNLMS  v  comparison. 
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Fig.  4.  AMPNLMS  £  comparison. 


mance,  however;  the  feasible  water-filling  algorithm  outperforms  the 
AMPNLMS  in  the  initial  convergence  stage.  The  AMPNLMS  and 
feasible  water-filling  algorithms  are  sensitive  to  the  choice  of  £  and 
a.  parameters. 

Note  the  appearance  of  a  significant  excess  MSE  in  the  case  of 
the  feasible  water  filing  algorithm.  This  originates  from  (a)  the  in¬ 
adequacy  of  (12)  for  approximating  E{zl{k)}  when  the  deviations 
are  small,  and  (b)  fluctuation  of  estimates. 

The  feasible  water  filling  algorithm  has  a  very  high  computa¬ 
tional  complexity  per  input  sampling  period.  The  increased  compu¬ 
tational  complexity  is  due  to  the  sort  operation  as  well  as  the  need 
to  compute  the  c  and  Q  elements.  In  contrast,  the  AMPNLMS  re¬ 
quires  five  multiplications,  one  addition,  and  a  square  root  operation 
(which  maybe  approximated)  beyond  the  MPNLMS  algorithm  [7]. 

7.  CONCLUSIONS 

We  have  proposed  two  new  feasible  PtNLMS  algorithms.  The  fea¬ 
sible  water  filling  algorithm  was  motivated  by  trying  to  find  the  op¬ 
timal  gain  at  each  step.  This  approach  resulted  initially  in  the  non- 
feasible  water  filling  algorithm.  We  subsequently  modified  the  non- 
feasible  version  into  a  suboptimal  feasible  version. 

Next  we  introduced  the  AMPNLMS  algorithm.  This  algorithm 
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Fig.  5.  Algorithm  comparison. 


was  based  on  allowing  /r-law  coefficient  compression  to  be  time- 
varying.  In  doing  so,  the  e-neighboorhood,  which  the  algorithm  is 
required  to  converge  to,  is  slowly  tightened  as  time  goes  by.  The 
tightening  is  controlled  by  the  evolving  MSE.  This  modification  re¬ 
sults  in  superior  MSE  convergence. 
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